@recris amazing work

did you notice this is solving issue of training multiple same class concept?

like 2 man at the same time

or when you train a man it makes all other mans to turn into you.

is this solving this problem

moreover, after training, you dont need to use redux right with vision_cond_dropout = 0.5 + vision_cond_ratio = 0.2

@recris amazing work

did you notice this is solving issue of training multiple same class concept?

like 2 man at the same time

or when you train a man it makes all other mans to turn into you.

is this solving this problem

moreover, after training, you dont need to use redux right with vision_cond_dropout = 0.5 + vision_cond_ratio = 0.2

This has nothing to do with either of those issues. For multiple concepts you would need something like pivotal tuning which currently is not supported either.

This PR is only an attempt to improve overall quality in the presence of poorly captioned training data.

@recris thanks but you still recommend vision_cond_dropout = 0.5 + vision_cond_ratio = 0.2 and then we can use trained lora without flux redux right?

Please read the notes fully before posting - these are not "recommendations", this hardly has been tested in a comprehensive way and it probably is not ready for widespread use.

That said, you can probably start with vision_cond_dropout = 0.5, vision_cond_ratio = 1.0. Beware that this could also require changes to the learning rate or total number of steps trained to achieve same results as before.

Interesting concept!
About the LoRA only learning the difference between (base model + image conditioning) and training data:

if you consider this a downside and want a stand-alone LoRA as output, you could try to (gradually?) remove the image conditioning from the model prediction, but still expect the model to have learned making the same prediction as if it was still conditioned. Similar to this concept:

Nerogar/OneTrainer#505

but not using the base model as teacher, but the base model conditioned by Redux.
A LoRA that replicates the Redux conditioning - but without Redux - could be the result, which could then be improved by regular training on data.

if you consider this a downside and want a stand-alone LoRA as output, you could try to (gradually?) remove the image conditioning from the model prediction, but still expect the model to have learned making the same prediction as if it was still conditioned

This is what the vision_cond_dropout can be used for - you feed the model a mix of caption conditioned plus Redux conditioned samples so it learns to not become dependent on Redux. A value of at least 0.5 seems to do the trick, but maybe you can even go lower.

After some additional testing, in my (subjective) experience I can set vision_cond_dropout as low as 0.25 and still get good results.

I still don't think mashing together text and vision conditioning is the right approach. The correct way to do this would require concatenating the embeddings, then use a non-binary attention mask to control the influence of the Redux tokens. But this would require deeper changes to the Flux model code.

if you consider this a downside and want a stand-alone LoRA as output, you could try to (gradually?) remove the image conditioning from the model prediction, but still expect the model to have learned making the same prediction as if it was still conditioned

This is what the vision_cond_dropout can be used for - you feed the model a mix of caption conditioned plus Redux conditioned samples so it learns to not become dependent on Redux. A value of at least 0.5 seems to do the trick, but maybe you can even go lower.

If I understand your code correctly, the dropout removes the Redux-conditioning. Therefore, in a dropout-step you are training normally against the training images.
Is that really helpful towards making the resulting LoRA work well without Redux? You are basically optimizing towards the training images under two conditionings independently from each other, with and without Redux - but then during inference Redux is not available.

I have now experimented myself a bit with this idea: On the left is the (only) training image, middle and right are samples of the LoRA, without Redux:

Only a few minutes of training each, 100-200 steps. The LoRA was trained with a Redux-conditioned teacher prediction, but was itself not Redux-conditioned - so there is no mismatch between training and inference.

Here is some experimental code: Nerogar/OneTrainer@master...dxqbYD:OneTrainer:redux

I am still running some tests; I had a configuration mistake which might have affected my earlier results.

With Redux dropout my expectation was that it would "diversify" the captions seen during training, improving learning robustness; the concept here is similar to providing multiple captions for the same image with varying levels of detail and selecting one at random in each training step (for reference: #1643)

I am still running some tests; I had a configuration mistake which might have affected my earlier results.

With Redux dropout my expectation was that it would "diversify" the captions seen during training, improving learning robustness; the concept here is similar to providing multiple captions for the same image with varying levels of detail and selecting one at random in each training step (for reference: #1643)

I get the idea, but what I am saying is:

you are doing "multi-caption training",
a) one of which is an actual caption and
b) the other one is an image embedding.

And then you are done with training, and during inference you can only use a). You cannot access what the model has learned for b) because you don't have the embedding.

That's why I'm proposing that there should only be one optimization target - the one you still have during inference. That's what I've done in my samples.

The central line in my code is this:
https://github.com/dxqbYD/OneTrainer/blob/7d0e8878316d477f00d46d512ad39c3d750f7e42/modules/trainer/GenericTrainer.py#L700

@dxqbYD I think what you're talking about is only valid for unknown things. For example, if we're training a specific person, then we naturally want to give the model his name manually. But for some known concepts, for example, training cars to be less broken, we don't need to manually tell the model that a car is a car. Siglip will do it for us perfectly.
In my tests, Redux really reduce efficiency of training exact concepts, but at the same time, some general details start to look better.

@dxqbYD I think what you're talking about is only valid for unknown things. For example, if we're training a specific person, then we naturally want to give the model his name manually. But for some known concepts, for example, training cars to be less broken, we don't need to manually tell the model that a car is a car. Siglip will do it for us perfectly. In my tests, Redux really reduce efficiency of training exact concepts, but at the same time, some general details start to look better.

sorry, I don't understand how any of this is related to what I wrote before.

I've changed my previous approach to controlling the strength of the Redux conditioning:

• Now it performs downsampling of the Redux tokens to a N by N grid controlled by the vision_cond_downsample parameter
• Text tokens are always used during training and Redux tokens are concatenated. Dropout feature still works as before.

The amount of Redux tokens in the conditioning seems to affect the ability to make the LoRA usable with text prompts, from testing various downsampling sizes I've noticed that beyond N=5 (25 tokens) it starts to perform noticeably worse (unless counteracted with dropout)

Also training with both text and Redux seems to have a negative performance impact due to the amount of tokens being used, but there is also a significant amount of padding being added by default. I recommend lowering t5xxl_max_token_length if not using very long captions.